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Abstract —We develop a method for evaluating restricted 
isometry constants (RICs). This evaluation is reduced to the 
identification of the zero-points of entropy density vrhich is 
defined for suhmatrices that are composed of columns selected 
from a given measurement matrix. Using the replica method 
developed in statistical mechanics, vre assess RICs for Gaussian 
random matrices under the replica symmetric (RS) assumption. 
In order to numerically validate the adequacy of our analysis, 
we employ the exchange Monte Carlo (EMC) method, which has 
been empirically demonstrated to achieve much higher numerical 
accuracy than naive Monte Carlo methods. The EMC method 
suggests that our theoretical estimation of an RIC corresponds to 
an upper hound that is tighter than in preceding studies. Physical 
consideration indicates that our assessment of the RIC could he 
improved hy taking into account the replica symmetry breaking. 

I. Introduction 

The signal processing paradigm of compressed sensing (CS) 
enables a substantially more effective sampling than that re¬ 
quired by the conventional sampling theorem HI. CS is applied 
to problems in various fields, in which the acquisition of data 
is quite costly, such as astronomical and medical imaging Q, 
0. The CS performance is mathematically analyzed using 
the problem settings of a randomized linear observation a, 
0. Here, A G is the given observation matrix, and CS 

endeavors to reconstruct the 5-sparse signal x G that has 
S{< N) nonzero components from observation y =Ax. 

A widely used strategy for the reconstruction of this signal 
is the ii minimization, which corresponds to the relaxed 
problem of Iq minimization. A key quantity used to analyze 
the £q and ii minimization strategies is the restricted isom¬ 
etry constant (RIC) 0. Literally evaluating an RIC requires 
the computation of maximum and minimum eigenvalues of 
N\/{S\{N — S)l) submatrices that are generated by extracting 
5-columns from A, which is computationally infeasible. In the 
case of Gaussian random matrices of A, the upper bound for 
the RIC is estimated using the large deviation property without 
direct computation of the eigenvalues 0, m, 0. 

This paper proposes a theoretical scheme for the direct 
estimation of the RICs. In order to do this, we evaluate the 
entropy density of the submatrices that provide a given value 
of the maximum/minimum eigenvalues. An RIC of matrix A 


is offered by the condition that the corresponding entropy 
vanishes. Eurthermore, in order to demonstrate our method’s 
utility, we apply our scheme to Gaussian random matrices, 
using the replica method, and compare the obtained result with 
that of earlier studies. 

Our theoretical evaluation is also numerically assessed using 
the exchange Monte Carlo (EMC) sampling a, which is 
expected to achieve much higher numerical accuracy than 
those of naive Monte Carlo schemes. The EMC method en¬ 
ables effective sampling, avoiding entrapment at local minima, 
which limits the effectiveness of naive Monte Carlo sampling 
to capture the true behavior QO). Numerical results suggest 
that our scheme currently provides the tightest RIC upper 
bound, which could be further tightened by taking into account 
the replica symmetry breaking (RSB). 

IT Restricted isometry constant 

In the following, we assume that VA G is normalized 

so as to (typically) satisfy (A^A),-,- = 1 for all iG {I,-- - ,N}. 

Definition 1 (Restricted isometry constants). A matrix A G 
■^MxN ihg restricted isometry property (RIP) with RIC 

0 < if 

(l-5r)||jc||2 <||Ajc||| <(l + 5r’‘)lkllf (1) 

holds for any S-sparse vector x G in which S is the number 

of non-zero components. 

The original work presented by Candes et al. a addresses 
symmetric RIC 5s = max[5™",5™“]. An RIC indicates how 
close the space, which is spanned by the 5-columns of A, 
is to an orthonormal system. If an RIC is small, the linear 
transformation performed using A is nearly an orthogonal 
transformation. 

The symmetric RIC provides sufficient conditions for the 
reconstruction of 5-sparse vector x in underdetermined linear 
system y = Ajc using Iq and £\ minimization 0|. 

Theorem 1. Let A G and x G R^ with M < N, and 

consider the linear equation y = Ax. If dzs < L a unique 5- 
sparse solution exists and is the sparsest solution to £o problem 

mm\\x\\o, subject to y = Ax. (2) 


Also, if &2S < 1, the S-sparse solution to problem 

\m\i\\x\\\, subject to y ~ Ax (3) 

X 

is uniquely identified as the sparsest solution and equals the 
(.Q problem’s solution. 

It should be noted that 5™" and 5™’' do not increase or 
decrease at the same rate, and asymmetric RICs improve the 
condition of reconstruction HD- 

Theorem 2. Consider the same problem settings as in Theo¬ 
rem [7] If (4-</2 — < 4 ( 1/2 — I), then the unique 

S-sparse solution is the sparsest solution to the problem 
and equals the solution to the io problem OB- 

RIC evaluation is also a fundamental linear algebra problem 
Q, ii because RICs clearly relate to the eigenvalues of Gram 
matrices. Let T <ZV = {I,--- ,A^}, iLj = 5 be the position of 
the nonzero elements of S'-sparse vector x. The product Ax 
equals AjXt, where Aj is the submatrix that consists of i GT 
columns of A and where xj = {xi\i G T}. For any realization 
of T, the following holds. 

An,in(AjA7-)||j:7’|||- < HArJCrU^ < Xmia{^A.T)\\XT\\F 

Here, Amin(fi) and Aniax(fi) denote the minimum and maxi¬ 
mum eigenvalues of B, respectively, and superscript T denotes 
the matrix transpose. Therefore, the following expression of 
the RIC is equivalent to eq. O: 

dr = I - A^„(A;5), 5^“ = - I, (4) 

in which 

A;i„(A;5)= min f^^iAlAr), (5) 

T-.T<ZV.\T\=S 

■^max (A; ‘5) = max Amax (A ^ A r). (6) 

T-.T<ZV.\T\=S 

Literal evaluation of eq. (|4]i requires the calculations of the 
maximum and minimum eigenvalues of the N\/ {S\{N — S)\) 
Gram matrices {A^A^}, which is computationally difficult 
when N and S are large. For typical Gaussian random matrices 
A, the RIC’s upper bound is estimated using large deviation 
properties of the maximum and minimum eigenvalues of the 
Wishart matrix 0, Q, i). 

III. Problem setup and formalism 

We estimate RICs in a different manner, and the following 
theorem is fundamental to our approach. 

Theorem 3. Let A G Then the minimum and maximum 

eigenvalues of A^A are given by 

■^mn(A’^A) = - lim ^logZ(A;j3), (7) 

p^+oc Nji 

Amax(A’^A) = - lim ^logZ(A;j3), (8) 

P^-ooI\p 

respectively, where Z(A;j3) is defined using u G K^.' 

Z{A-,I5) = Jdue^W^^WcSdluWl-N). (9) 


Proof: Applying identity 5(||i<|||- — N) = j5/{4n)x 
Jdriexp(^—l5r]/2{\\u\\p—N)) gives us 


Z(A;j3) = 


( 2 ;r)T 


2)3 f 


rh 


exp 


L^l 


(Nri I _ 


in which {A,} is the /th eigenvector of A^A. As )3 —> + 0 °, 
the integral can be evaluated using the saddle point method, 
which is dominated by 77 = —Amin(A^A) + +o()3^*), 

where o()3^*) represents the contribution from negligible 
terms compared with j3^/ This yields eq. 0, and eq. 
is similarly obtained by applying the saddle point method for 

j3 —>■ □ 

Theorem [ 3 ] holds for all submatrices Aj. For mathematical 
convenience, we introduce variables c G {0, l}^ and define 


Zc(c,A;fi) = JduP{u\c)expi^-^\\A{cou)\\p'^ 


X 5 (||com||/-A), 


(lO) 


where o denotes the component-wise product, and P{u\c) <== 
expf-I^Li(l -Ci)uj is introduced in order to avoid 
the divergence caused by integrating iq when Ci = 0. Let us 
define c(T) G {0, l}^ to be (c(7’)), = I for i G T and to 
be {c{T))i = 0 otherwise. The two functions Z{AT',ji) and 
Zc{c{T),A-,fi) have a one-to-one correspondence: Z{At',I 5) = 
Zc{c{T),A-,fi). We write Aniax(c,A) and Aniin(c,A), which 
are obtained by substituting Zc{c,A-,fi) into eq. 0 and 
eq. ®, respectively. Because Aniax(A 7 'A 7 ') — Aniax(^(T),A) 
and Amii/A^^Aj’) = Aniin(c(r),A) naturally hold, eqs. (|5]l6]l can 
be respectively rewritten as 


A;|;in(A;5) = min Amin (c, A), 

cecs 

A*ax(A;5) = maxAmax(c,A), 
cecs 


(11) 

( 12 ) 


where Cs denotes the set of configurations of c that satisfy 
Lq = 5. 

We hereafter focus on the situation in which both M and S 
are proportional to A as M = Na and S = Np, respectively, 
where a,p ~ (9(l). Let us define the energy densities of c to 
be A+(c|A)=Amin(c,A)/2 and A_(c|A)=Amax(c,A)/2. Based 
on this, we introduce a free entropy density as 0(/i|A;p) = 
A^^og -^P) 1 where sgn(p) de¬ 

notes the sign of p. Eqs. 0[8]) offer its alternative expression 


0 (p|A;p)= limA 'log[^z/(c,A;^) 5 (^q 

4^+00 C ^i=l 


i-Np 


(13) 


In addition, we represent the number of c that correspond to 
A±(c|A) = XI2 and satisfy Y.iCi=Np as exp(Aw±(A|A;p)) 
using entropy densities a)±(A|A;p), which are naturally as¬ 
sumed to be convex functions of A. Summation over the 
microscopic states of c is replaced with the integral of A over 
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Fig. 1. Schematic picture of entropy curve and relationship to parameter 


the possible value of A±(c|A): 


0(M|A;p) = ^log 


JdX exp{-Ap A +A^a)sgn(^) (A |A; p)} 

(14) 


■ max{-pl + a)sgn{^)(A|A;p)}, 

A 


in which the saddle point method is employed. The maximizer 
of X, which corresponds to the typical energy value of c that is 
sampled following the weight c,-— Ap), 

must satisfy 


^ <3tl)sgn(^)(A|A,p) 


(15) 


Eq. (fT4li implies that 0(p|A;p) is obtained using the Legen¬ 
dre transformation of ( 0 ±{X\A-,p), and the inverse Legendre 
transformation converts 0(p|A;5) to a)±(A|A;p) as 

®sgn(/i)(^|A;p) = ^(p|A;p)-p^^^^^^, ( 16 ) 

from the convexity assumption of a)±(A|A;p). A similar for¬ 
malism has been introduced for investigating the geometrical 
structure of weight space in learning of multilayer neural 
networks Ha. 

The relationships among p, X, and (0± are illustrated in 
Fig.IU Entropy densities co+ and co are convex increasing and 
decreasing functions of X, respectively. According to eq. 
the value of A at p represents the point where the gradient of 
a)± equals p. By definition, negative entropy values are not 
allowed, and a)±(A|A;p) < 0 implies that no c simultaneously 
satisfies both A±(c|A) = A/2 and =Np. Therefore, the 
Ai^ that produces a)±(A)^^4;p)=0 is the possible minimum or 
maximum eigenvalue. Hence, eqs. (II llll2t give us 


A;i„(A;p)=A;, X,l,,{A-,p)=Xl, (17) 


which are the typical values for p = Pmax and p = Pmim 
respectively (Lig. [Til. 
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Fig. 2. Entropy curve for a = 0.5 and p = 0.1 with (a) p > 0 and (b) p < 0. 
Circles denote EMC method results. Vertical lines represent (a) minimum and 
(b) maximum eigenvalues of MP distribution. 


IV. RS ANALYSIS FOR GAUSSIAN RANDOM MATRIX 

This section applies the methodology introduced in the 
previous section to the case in which components of A are 
independently generated using a Gaussian distribution with 
mean 0 and variance (Aa)^T In this case, 0(p|A;p) and 
a)±(A|A;p) randomly fluctuate depending on A. However, 
for all e > 0, the probability that deviation from the typical 
values, (j){p-,p)=[(j){p\A-,p)]A and cu±(A;p) = [a)±(A|A;p)]A, 
is larger than e tends to vanish as N ^ Here, [•].4 denotes 
the average of A. Therefore, typical properties can be charac¬ 
terized by evaluating the typical values, 0(p;p) and (0±{X), 
using the replica method with the identity 03, Ql: 

[log/(A)]A = lim^log[/"(A)]A (18) 

«->0 an 

where /(A) is an arbitrary function. When both n and m = p/ji 
are positive integers, regarding Q ~-Vp) 

in eq. (fljl l as /(A) leads us to express [/”(A)]a as a summa¬ 
tion/integration with respect to n and nm replica variables {c"} 
and {c" (a g {1,2,(7 G (1,2,...,/«}), which can 

be evaluated by the saddle point method for N ^ 

Under the replica symmetric (RS) assumption, in which the 
dominant saddle point is assumed to be invariant against any 
permutation of the replica indices a and a within each of their 
sets {1, 2 , ...,u} and ( 1 , 2 ,..., m}, respectively, the resulting 
































functional form of log[/''(A)]^ becomes extendable for 
non-integer ii and m. Therefore, we insert the expression into 
eq. (fT3]) employing the formula of eq. (fTSl) . which finally yields 


0(At;P) = -^^og{a + x + ^i{l-q)} + ^iog{a + x) 


ajlq 


g-|(l + i)+l2i + Kp 

2 2 \ iiJ 2 


2{a + x + ^iil-q)} 

j Dzlog 11 + j Dy exp ^ 


{■s/q\-qoy+'/^zf 

22 


)}. (19) 


where {q,x^Q:qOTq\^K} are determined to extremize the right 
hand side, and /Dz = -^= exp(—z^/2). The derivation of 

eq. (O is shown in Appendix lAl Entropy densities (0±{X',p) 
are derived by applying the inverse Legendre transformation 
to 0(p;p). 


V. Results 

In Fig. 121 entropy densities a)± with a = 0.5 and p = 0.1 
are shown for (a) p < 0 and (b) p > 0. Results of the exchange 
Monte Carlo (EMC) sampling are represented by circles, and 
the EMC procedure is summarized in Appendix iBl 

The values of A when p —+0 and p ——0, which 
are denoted using dashed lines, coincide with the respective 
minimum and maximum of the Marchenko-Pastur (MP) distri¬ 
bution’s support for the M x S Gaussian random matrix ifTSll . 
As the limit of |p| —?► 0 corresponds to unbiased generation 
of M X 5 Gaussian random matrices, the coincidence theo¬ 
retically supports the adequacy of our analysis. The slight 
discrepancy between the theoretical and EMC results in the 
entropy’s tails could be due to the insufficiency of the RS 
assumption. The convexity of our entropy suggests that the 
RS assumption exactly creates the entropy curve or extends it 
outward ESI. This is consistent with the EMC method’s result, 
which indicates that the exact entropy curve is inward when 
compared to that produced by the RS assumption. Therefore, 
the estimated zero-points, and that are provided 

using the RS assumption, are meaningful upper and lower 
bounds, respectively, of the true values. We call them RS 
bounds. 

Fig. |3] compares our RS upper bound. Bah and Tanner’s 
upper bound il, and the RIC numerically obtained lower 
bound EH- In this example, the symmetric RIC is 5™’'. Our 
analysis lowers the upper bound of the RIC, especially for 
a large p/a region. Over the entire parameter region, our 
estimates are consistent with the numerically obtained lower 
bound. 

Fig. ID shows the parameter region that mathematically 
supports Iq and £i reconstruction according to Theorems 1 
and 2. The region determined by the Bah and Tanner RIC is 
indicated using black lines. The RS bound of the RIC extends 
the region in which correct reconstruction is guaranteed, and 
further extension may be provided by taking the RSB into 
account. 



Fig. 3. Comparison of symmetric RIC for a = 0.5. Numerical lower bound 
is estimated for N = 1000 and M = 500. 



Fig. 4. £q and li limits given by RS RIC. Black lines represent Bah and 
Tanner’s results, denoted by BT. 


VI. Summary and conclusion 

We proposed a theoretical scheme for the evaluation of 
restricted isometry constants. The problem was converted to 
the assessment of entropy density, and the possible maximum 
and minimum eigenvalues, which produce the RIC, are the 
entropy’s zero-points. Given a Gaussian random matrix, we 
computed the entropy density using the replica method under 
the replica symmetric ansatz and estimated the value of the 
RIC. Physically, it has meaning as a bound and is tighter 
than existing bounds. Numerical experiments using the EMC 
sampling support our analysis. 

A more accurate evaluation of the RIC is possible if the 
RSB is taken into account. Our scheme is applicable to more 
general matrices than Gaussian random matrices as well. 

Appendix A 

RS CALCULATION OF FREE ENTROPY DENSITY 

Identities 

1 = f , (20) 

J V -^,= 1 2 














for all combinations of replica indices (a, O') 
and (i’,T) {a,b = 1,2,...,«;(7,t = 1,2,... ,m), 

are employed in the saddle point assessment of 

= iV->log[(I,Z'"(c,A;j3)5(ltiC,- - iVp))"U. 

We assume that the dominant saddle point is of the replica 
symmetric form as 

{ 1 for a = b, O = T 
qi for a = b, O (21) 

qo for a ^ b. 

This means that when A is a Gaussian random matrix of mean 
0 and variance 

S^iviSabSat + qiSabi^ - 5CTT)+g'o(l “ Sab)) 

holds, where Higher order correlations are 

negligible due to the central limit theorem, which indicates 
that can be expressed as —qiw‘lf + 

\/q\ - qQvfi + y^z^), where w"^, v", and are i.i.d. 
Gaussian random variables of zero mean and unit variance. 
Replacing [•],4 with average with respect to these Gaussain 
variables, the saddle point evaluation offers an expression of 
^j3(ot,p) = lim„^o^0j3(«,'«;p), as 


, / m{Q+qiqi)-m^{qiqi-qoqo) f „ 

{m;p ) =- - - +J Dzlog^p (z) 


+ a 


_Mo_■ 

2{a + l5{l-qi)+p{qi-qo)}- 


M(gi-go) \ 

a+p{l-qi)) 

+ Kp, (22) 


where 


mW = i+ 


^-K 


(Q + qi)"^/^ 


jDyexp(^ 


m{^/ql-qoy+^/^z)^ 

2iQ + qi) 


and Q, K, qi and qo are conjugate variables for the integral 
representations of delta functions in eq. (fTOl i. eq. (fOl) and 
eq. (l20l i. respectively. Eq. (l22l) yields the free entropy density 
as 0(p;p) = limj 3 ^„0p(p/j3,p), in which the variables scale 
so that Q = m{Q + qi), q\ = rn^qi, qo = rn^qo, and j = j3(l — 
^i) become 0(1). This gives the expression of eq. (fT^ . 

The variables are determined by extrem- 

ization conditions of the free entropy density eq. (US, 



q = 

1 = 

p = 


I 
I 




S(z) Vg(g-A) 


S(z) 


+ 


qoz^ \ 
iQ-A)^) 


A = 


qo = 


ap^i^-q) 

{a + x){a + x + p{i-q)} 

ap^q 

{a + x + Pii-q)V 


(23) 

(24) 

(25) 

(26) 

(27) 

(28) 


where q = qo, A^qi- qo, and E(z) = lim|3^„ (z). 


Appendix B 

Monte Carlo sampling for RIC estimation 

We employ the exchange Monte Carlo (EMC) sampling 
El in order to numerically compute the free entropy density 
^(p|A;p) and obtain the entropy density avoiding the trap of 
metastable states. In the EMC approach, we prepare k systems, 
which have the same configuration of A, and assign configura¬ 
tion c, € Cs and parameter p, to each system i= 1, • • • ,k. The 
signs of {p,} are set to be the same. Each step of the EMC 
process updates c, within each system, and attempts exchanges 
between configurations c, and c,+i. The probability of transi¬ 
tion from Ci to c\ is given by w{ci,c\) = min{exp(p,WA,), 1}, 
where A; = Asgn(^.)(c,|A) - Asgn(^^)(c(|A). The probabil¬ 
ity of an exchange between systems c,- and c,+i is 
given by Wexc(G,c,+i) = min{exp(A(pi - p,+i)A,i+i), 1}, 
in which A,g+i = Asgn(^.)(c,-|A) - Asgn(^_^,)(Ci+i|A). Af¬ 
ter sufficient updates, the entire A:-system is expected 
to converge to equilibrium distribution Ptot({c,}|A) £>= 

nti exp{-A^pAgn(M,) (gI> 1 )}- 

The density of states W±{X\A',p)=Y,cecs S{X/2 — A±{c\A)) 
is obtained by applying the multihistogram method ifTSll using 
histgrams of A± obtained by EMC sampling. Einally, the free 
entropy density is calculated: 

0(p|A;p) = y ^flWsgn(^)(A|A;p)exp(-Apl/2), (29) 

and the entropy density is derived by applying the inverse 
Legendre transformation to eq. (|29]) . 

Acknowledgment 

The authors would like to thank Tomoyuki Obuchi for his 
helpful comments and discussions. This work was partially 
supported by the RIKEN SPDR fellowship and by KAKENHI 
No. 26880028 (AS), and KAKENHI No. 25I200I3 (YK). 

References 

[1] H. Nyquist, Trans. AIFE 47, 617 (1928). 

[2] D. D. Lustig and J. Pauly, Magnet. Reson. Med. 58 (6), 1182-1195 
(2007). 

[3] J. Bobin et al., IEEE J. Sel. Top. Signal Process. 2 (5), 718-726 (2008). 

[4] E. Candes et al., IEEE Trans. Inform. Theory 52 (2), 489-509 (2006). 

[5] D. Donoho, IEEE Trans. Inform. Theoiy 52 (4), 1289-1306 (2006). 

[6] E. Candes and T. Tao, IEEE Trans. Inform. Theory 51 (12), 4203—4215 
(2005). 

[7] J. D. Blanchard et al., SIAM Rev. 53 (1), 105-125 (2011). 

[8] B. Bah, and J. Tanner, SIAM J. Matrix Anal. & Appl. 31 (5), 2882-2898 
( 2010 ). 

[9] K. Hukushima, and K. Nemoto, J. Phys. Soc. Jpn, 65, 1604—1608 (1996). 

[10] D. Donoho and Y. Tsaig, Signal Process. 86 (3), 549-581 (2006). 

[11] S. Eoucart and M.-J. Lai, Appl. Comput. Harmon. Anal. 26 (3), 395^07 
(2009). 

[12] R. Monasson and D. O’Kane, Europhys. Lett. 27 (2), 85-90 (1994). 

[13] M. Mezard et al.. Spin Gla.ss Theory and Beyond, (World Sci. Pub., 
1987). 

[14] H. Nishimori, Statistical Physics of Spin Glasses and Information 
Processing: An Introduction, (Oxford Univ. Pr., 2001). 

[15] V. A. Marchenko and L. A. Pastur, Mat. Sb. (N. S.) 72 (114), 507-536 
(1967). 

[16] T. Obuchi et al., J. Phys. A: Math. Theor. 43, 485004 (28pp) (2010). 

[17] C. Dossal et al.. Linear Algebra Appl. 432, 1663-1679 (2010). 

[18] A. M. FeiTenberg and R. H. Swendsen, Phys. Rev. Lett. 64, 1195-1198 
(1989). 




















